Search for: All records

Abstract Partially Observable Markov Decision Processes (POMDPs) can model complex sequential decision-making problems under stochastic and uncertain environments. A main reason hindering their broad adoption in real-world applications is the unavailability of a suitable POMDP model or a simulator thereof. Available solution algorithms, such as Reinforcement Learning (RL), typically benefit from the knowledge of the transition dynamics and the observation generating process, which are often unknown and non-trivial to infer. In this work, we propose a combined framework for inference and robust solution of POMDPs via deep RL. First, all transition and observation model parameters are jointly inferred via Markov Chain Monte Carlo sampling of a hidden Markov model, which is conditioned on actions, in order to recover full posterior distributions from the available data. The POMDP with uncertain parameters is then solved via deep RL techniques with the parameter distributions incorporated into the solution via domain randomization, in order to develop solutions that are robust to model uncertainty. As a further contribution, we compare the use of Transformers and long short-term memory networks, which constitute model-free RL solutions and work directly on the observation space, with an approach termed the belief-input method, which works on the belief space by exploiting the learned POMDP model for belief inference. We apply these methods to the real-world problem of optimal maintenance planning for railway assets and compare the results with the current real-life policy. We show that the RL policy learned by the belief-input method is able to outperform the real-life policy by yielding significantly reduced life-cycle costs. 

Structural health monitoring of complex structures is often limited by restricted accessibility to locations of interest within the structure and availability of operational loads. In this work, a novel output-only virtual sensing scheme is proposed. This scheme involves the implementation of the modal expansion in an augmented Kalman filter. Performance of the proposed scheme is compared with two existing methods. Method 1 relies on a finite element model updating, batch data processing, and modal expansion (MUME) procedure. Method 2 employs a recursive sequential estimation algorithm, which feeds a substructure model of the instrumented system into an Augmented Kalman Filter (AKF). The new scheme referred to as Method 3 (ME-AKF), implements strain estimates generated via Modal Expansion into an AKF as virtual measurements. To demonstrate the applicability of the aforementioned methods, a rollercoaster connection was instrumented with accelerometers, strain rosettes, and an optical sensor. A comparison of estimated dynamic strain response at unmeasured locations using three alternative schemes is presented. Although acceleration measurements are used indirectly for model updating, the response-only methods presented in this research use only measurements from strain rosettes for strain history predictions and require no prior knowledge of input forces. Predicted strains using all methods are shown to sufficiently predict the measured strain time histories from a control location and lie within a 95% confidence interval calculated based on modal expansion equations. In addition, the proposed ME-AKF method shows improvement in strain predictions at unmeasured locations without the necessity of batch data processing. The proposed scheme shows high potential for real-time dynamic estimation of the strain and stress state of complex structures at unmeasured locations. 

High cycle fatigue is a major cause of cracking in steel structures subjected to cyclic loading. It can result in substantial financial losses and structural failures compromising the safety of users. Uniaxial methods are in many cases insufficient for large in-service structures with complex geometry and connections subjected to multiaxial non-proportional loadings. A new method for fatigue life prediction for complex structures is presented using the critical plane method and the Kalman filter. The applicability of the methodology proposed is demonstrated and evaluated in a roller coaster support structure. Strain rosettes and accelerometers were installed on a support bracket near weld lines to measure responses. A substructure model is defined and used to estimate response prediction in the weld of the support bracket. The estimation of the input and the state estimation is performed using the augmented Kalman filter method, based on the response measurements and the substructured model. This new methodology is anticipated to be used for real-time fatigue prognosis of highway bridges.